Image recognition device, image recognizing method, storage medium that stores computer program for image recognition

ABSTRACT

An image identifying device includes: a setting unit which sets a section having at least one image in a video; a first recognizing unit which calculates a plurality of feature amounts related to at least the one image and which acquires a plurality of identification results corresponding to each of the feature amounts from an identifier which may identify a plurality of objects belonging to a first category; a selecting unit which selects, based on the identification results, a second category of a third category ; and a second recognizing unit which calculates another feature amount related to an image included in another section and acquires another identification result corresponding to the feature amount from another identifier which may identify the objects included in the second category.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is based upon and claims the benefit of priority fromthe prior Japanese Patent Application No. 2011-146077 filed on Jun. 30,2011, the entire contents of which are incorporated herein by reference.

FIELD

The embodiments discussed herein are related to an image recognitiondevice that recognizes, for example, an object included in a video, animage recognizing method, and a storage medium that stores a computerprogram for image recognition.

BACKGROUND

It is preferable that a label indicating a name of various objectsincluded in a video is added to the video. For example, if theabove-described label is attached to a television program image, abroadcast station may search desired images if necessary from among manytelevision program images. However, manually attaching the labels to thevideo takes huge amount of time and effort.

In recent years, a technique for automatically recognizing the name ofthe object included in the image has been studied. For example, there isa technique disclosed in T. Malisiewicz and A. A. Efros, “Recognition byassociation via learning per-exemplar distances”, Proceedings of theIEEE Computer Society Conference on Computer Vision and PatternRecognition (CVPR), 2008, p. 1-8. If it is possible to automate theoperation for adding labels to videos by using the above-describedtechnique, the operation for adding the labels is efficient.

On the other hand, a technique for adding additional information to thevideo according to the content of the video has been proposed. Forexample, there is a technique for adding the program information relatedto each program broadcasted by digital broadcast and the genre of theprogram to the video. Furthermore, for example, there is a technique forextracting a frame in which the telop character is displayed from theimage and recognizing the telop character. With reference to thedatabase in which the font of the character corresponds to the imagetype, the technique determines the image type corresponding to the fontof the recognized telop character. The above-described techniques aredisclosed in, for example, Japanese Laid-open Patent Publication No.2003-134412 and Japanese Laid-open Patent Publication No. 2006-53802.

SUMMARY

According to an aspect of the invention, an image identifying deviceincludes: a setting unit which sets a section having at least one imagefrom among a plurality of images included in a video; a firstrecognizing unit which calculates a plurality of feature amounts relatedto at least the one image and which acquires a plurality ofidentification results corresponding to each of the feature amounts froman identifier which may identify a plurality of objects belonging to afirst category; a selecting unit which selects, based on theidentification results, a second category having some objects from amongthe plurality of objects belonging to the first category instead of athird category having other objects with at least one object differentfrom the some objects; and a second recognizing unit which calculatesanother feature amount related to an image included in another sectiondifferent from the section and acquires another identification resultcorresponding to the feature amount from another identifier which mayidentify the objects included in the second category.

The object and advantages of the invention will be realized and attainedby means of the elements and combinations particularly pointed out inthe claims. It is to be understood that both the foregoing generaldescription and the following detailed description are exemplary andexplanatory and are not restrictive of the invention, as claimed.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram illustrating image recognizing processing performedby an image recognition device according to an embodiment;

FIG. 2 is a diagram illustrating a configuration of an image recognitiondevice according to an embodiment;

FIG. 3 is a functional block diagram of a processing device; and

FIG. 4 is an operation flowchart of the image recognizing processing.

DESCRIPTION OF EMBODIMENTS

The conventional technique uses information that is added to a video inadvance. Therefore, the embodiment is not applicable if the informationis not added to the video in advance. According to the technique forrecognizing the object included in the image, an object recognizer thatrecognizes all objects that are assumed to be included in the image isused. Actually, however, in most cases, simply some parts of the objectsto be recognized by the object recognizer are included in the video.Since the object identifier is formed to recognize many objects, theconventional object identifier may cause wrong recognition. The wrongrecognition means that the object identifier wrongly recognizes that theobject is included in the video even though the object is not actuallyincluded in the video.

The technique disclosed in the following embodiments may improve therecognition accuracy of the object included in the video.

With reference to the diagrams, an image recognition device according toan embodiment will be described. The image recognition device recognizesan object included in each image included in the video. FIG. 1 is adiagram illustrating image recognizing processing that is executed bythe image recognition device. In FIG. 1, the object, which is includedin images 101 and 102 included in a section 110 as a part of the inputvideo 100 is recognized by using a general object identifier 120 thatrecognizes all the objects A to Z included in a prescribed category. Thepredetermined category is a type of first category. For example, anobject A is recognized from the image 101, and an object B is recognizedfrom the image 102. Based on the recognized objects A and B, anindividual object identifier 123 that identifies simply an objectbelonging to a sub-category is selected from the individual objectidentifiers 121 to 124. The object that is to be recognized by theindividual object identifier is limited. The sub-category is a categoryto which the object included in each image included in the video 100 isestimated to belongs. The sub-category is a type of second category andthird category. By using the individual object identifier 123, theobjects included in the images 103 to 105 included in a section 111 ofthe rest of the video are recognized. By recognizing the object includedin the images 103 to 105 by using the individual object identifier 123,a possibility for wrongly recognizing an object that may not be includedin the images 103 to 105 is reduced. As a result, the recognitionaccuracy of the object is improved.

The image may be either a frame or a filed. The frame is a single stillimage included in the video. On the other hand, the field is a stillimage obtained by acquiring simply the data in even numbered rows or inodd numbered rows from the frame. The video includes a plurality ofimages time-sequentially arranged.

FIG. 2 is a diagram illustrating a hardware configuration of an imagerecognition device according to an embodiment. An image recognitiondevice 1 includes an input/output device 11, a storage device 12, aprocessor 13, and a storage medium reading device 14. The imagerecognition device 1 recognizes the object included in the videoobtained through the input/output device 11.

The input/output device 11 includes a video signal interface, which isused to couple the image recognition device 1 to a video input device(not illustrated) such as a camcorder or an image display device (notillustrated) such as a liquid crystal display, and a control circuitthereof. Alternatively, the input/output device 11 may include acommunication interface, which is used to couple the image recognitiondevice 1 to a communication network in accordance with a communicationstandard such as Ethernet, and the control circuit thereof. Theinput/output device 11 obtains a video from the video input device orthrough the communication network and then sends the video to theprocessor 13.

The input/output device 11 receives information related to the objectrecognized from the video from the processor 13. The input/output device11 outputs the information related to the object recognized from thevideo to the image display device such as a liquid crystal display.Alternatively, the input/output device 11 may transmit the informationabout the object recognized from the video to another apparatus coupledto the image recognition device 1 through the communication network.

The storage device 12 includes, for example, a semiconductor memory thatis volatile or non-volatile. The storage device 12 stores the computerprogram executed by the image recognition device 1, various parametersused to recognize an object from a video, and the like. The storagedevice 12 may store the obtained video. The storage device 12 may storeinformation, such as a type and a name of the object recognized from thevideo, with the video.

If a storage medium 15 is inserted, the storage medium reading device 14reads and sends the information or the computer program stored in thestorage medium 15 to the processor 13. Alternatively, the record mediumreading device 14 writes the information received from the processor 13into the storage medium 15. The storage medium 15 is a semiconductormemory, a magnetic disk, or an optical disk. For example, if the storagemedium 15 is a semiconductor memory, the storage medium reading device14 includes an interface circuit that communicates with thesemiconductor memory, such as, for example, an interface circuitcompliant with Universal Serial Bus. If the storage medium 15 is anoptical disk, the storage medium reading device 14 includes a drivedevice of the optical disk. The information stored in the storage medium15 is, for example, various parameters used to recognize an object froma video or a video as a target of the image recognizing processing.

The processor 13 includes one or several processors, a memory circuitsuch as a random access memory, and a peripheral circuit. The processor13 recognizes the object included in the video. Furthermore, theprocessor 13 controls the whole image recognition device 1.

FIG. 3 is a functional block diagram of the image recognition device 1.The image recognition device 1 functions as a setting unit 21, a firstrecognizing unit 22, a selecting unit 23, a calculating unit 24, adetermining unit 25, and a second recognizing unit 26. Each processor isa function module that is mounted according to a computer programexecuted on a processor included in the processor 13. Alternatively,each processing unit may be mounted on the image recognition device 1 asa separate calculation circuit. Alternatively, the units of theprocessor 13 may be mounted on the processor 13 as a one integratedcircuit by obtained by integrating the circuits corresponding to theunits.

By setting an inspection section as a target of the processing by thefirst recognizing unit 22, the setting unit 21 selects an image to beprocessed. The inspection section is a time interval that is set toinclude at least one image. For example, the setting unit 21 firstlysets a predetermined section that is set in advance to start from avideo as an inspection section. If the determining unit 25 determinesthat the inspection section is inappropriate, the setting unit 21extends the inspection section by a predetermined additional section.The determining unit 25 will be described below in detail.

For example, the length of the predetermined section is set to 1/100 to1/10 of the time length of the whole video. The length of thepredetermined section may be set to a period, for example, 1 to 10minutes regardless of the time length of the whole video. Furthermore,the predetermined section may be set to include the end of the videoinstead of the beginning of the video. Alternatively, the predeterminedsection may be set to start at a predetermined time from the beginningof the video, for example, the elapsed time of 5 to 30 minutes.Moreover, the predetermined section does not have to be one sequentsection. For example, the predetermined section is set to include aplurality of images arranged in an interval of 1 to 10 minutes.

The length of the additional section is set to 1/100 to 1/20 of the timelength of the whole video, for example. The length of the additionalsection may be, for example, 30 seconds to 5 minutes regardless of thetime length of the whole video. When setting the inspection section, thesetting unit 21 stores the time indicating the beginning of theinspection section and the time indicating the end of the inspectionsection in the storage device 12 and also reports the times to the firstrecognizing unit 22. Hereinafter, the section other than the inspectionsection in the video is referred to as a non-inspected section.

By using a general object identifier that recognizes all the objectsbelonging to a predetermined category with respect to at least one imagein the inspection section, the first recognizing unit 22 recognizes theobject included in the image.

At this time, the predetermined category is determined in advanceaccording to a purpose of information related to the object recognizedfrom the video, for example. For example, the predetermined category isdetermined to categorize the videos into genres such as “sports” and“drama” and to include all the objects that are useful to retrieve aspecific scene. Alternatively, the predetermined category may bedetermined to include simply the object that has a possibility ofappearing in a video of a specific genre.

The object according to the present embodiment may include not simplyobjects such as “person” and “car” occupying a specific range in a realspace but also “sky” that does not have a clear range. The objectaccording to the present embodiment may include a character or a figuresuch as “alphabet,” or “rectangular.” Moreover, the objects such as “redcar” and “blue car” that belong to the similar category and havedifferent features that are identifiable on the image may be individual.

Regarding each of the images, the first recognizing unit 22 recognizesthe object according to the following procedures. The first recognizingunit 22 divides one image in the inspection section into a plurality ofregions. The first recognizing unit 22 extracts at least one featureamount indicating the feature of the object to be recognized from theregions, respectively. By inputting the extracted feature amount into ageneral object identifier for each region, the first recognizing unit 22recognizes the object included in the region.

Specifically, the first recognizing unit 22 divides the image into aplurality of regions so that each object included in the image isincluded in the regions, respectively. Therefore, the first recognizingunit 22 has an integral pixel as one region in which the pixel value iswithin a predetermined range, for example. The first recognizing unit 22has an integral pixel as another region in which the pixel value is notwithin the predetermined range. In this case, the predetermined range isset in advance according to an assumed object, for example. The imagepixel value may be any of the color component values of RGB colorsystem, for example. Alternatively, the first recognizing unit 22converts the color component value of each pixel of the image into avalue of HSV color system or HLS color system, and a color phase, achroma, or a luminance value in the HSC color system or the HLS colorsystem may be a pixel value.

The first recognizing unit 22 couples the pixel with the differencebetween the target pixel and the pixel value is within the predeterminedrange from among the pixels adjacent to the target pixel in the image.For example, the pixel of the upper left end of the image is employed asa target pixel. The predetermined range is set to have the maximum valueof the difference of the image pixels that form an image of a object.The first recognizing unit 22 repeats the similar processing on thepixel coupled to the target pixel as another target pixel. The firstrecognizing unit 22 has a pixel assembly as a first region. By settinganother target pixel from among the pixels that are not included in thefirst region and repeating processing that is equivalent to theabove-described processing, the first recognizing unit 22 may divide theimage into a plurality of regions.

Alternatively, the first recognizing unit 22 may divide the image into aplurality of regions according to other various methods for dividing theimage into the plurality of regions. For example, the first recognizingunit 22 categorizes the pixel values of the pixels in the image into aplurality of clusters according to a clustering method such as k-meansmethod. The first recognizing unit 22 has an integral pixel belonging toeach cluster as a single region. After dividing the image into theplurality of regions, the first recognizing unit 22 may performexpansion and contraction calculation of morphology or contraction andexpansion calculation on at least one region from among the plurality ofregions to solve the isolated point. By performing labeling processingon each region, the first recognizing unit 22 may have each sub-regionas an individual region if a plurality of sub-regions that is separatedfrom each other.

The first recognizing unit 22 extracts at least one feature amount fromamong the plurality of regions of the image. Regarding the target regionfrom among the regions, the first recognizing unit 22 acquires a colorhistogram, as a feature amount, that indicates an appearance frequencyfor each of the color components. Alternatively, the first recognizingunit 22 may extract the feature amount that is determined based on aform or texture of the target region. For example, the first recognizingunit 22 may calculate the pixel included in the target region, and alength-to-width ratio or circularity of the target region as the featureamount. Based on the area S of the target region and the circumferencelength L of the target region, the circularity is expressed as (4πS/L²).In this case, S means a pixel included in the target region. The firstrecognizing unit 22 may have a wavelet coefficient as a feature amountobtained by wavelet transforming the target region. Further, the firstrecognizing unit 22 may calculate a Haar-like feature amount or aHistograms of Oriented Gradients (HOG) feature amount in the targetregion. Moreover, the first recognizing unit 22 may acquire thestatistic of the pixel value in the target region, for example, such asdispersion of color components, a difference between the minimumluminance value and the maximum luminance value or the average luminancevalue as a feature amount.

For each region of the plurality of images, the first recognizing unit22 inputs the feature amount vector based on the acquired feature amountinto the general object identifier. A feature vector is a vector with asingle element as each feature amount. The general object identifierrecognizes the object included in each region based on the input featureamount vector.

The general object identifier is a machine learning system, which isconfigured by so-called learning with teacher, such as the multilayerperceptron, the support vector machine, or the k-nearest neighboralgorithm. To make the general object identifier learn the features ofthe objects, the plurality of sample images are prepared in advance forall the objects belonging to predetermined categories. It is preferablethat a plurality of sample images that includes no object to berecognized is prepared. For each of the sample image, the same type ofthe feature amount of the type that is equivalent to the feature amountextracted by the first recognizing unit 22. The feature amount vectorwith the feature amount as an element extracted from the sample image isinput into the general object identifier, the general object identifierlearns by using a learning method according to a machine learning systemfor forming the general object identifier.

For example, if the general object identifier is a multilayerperceptron, the general object identifier learns according to a learningmethod called backpropagation. The general object identifier may includea plurality of support vector machines prepared for each object to berecognized. Basically, the support vector machine is a two-classidentifier. Therefore, each of the support vector machines learns byusing, for example, a method called kernel trick in such a way that thedetermining result, which indicates whether the object to be recognizedis not included, is output according to the input feature amount vector.

The parameter indicating the machine learning system that forms thegeneral object identifier, the weighting between the units and the biasof the units included in the multilayer perceptron, or each supportvector of the support vector machine is stored in advance in the storagedevice 12.

If the first recognizing unit 22 inputs the feature amount vector intothe general object identifier according to each of the regions of theimage, the general object identifier outputs the value indicating thetype of the object included in each of the regions. If the object to berecognized is not included in the region, the general object identifieroutputs the value indicating the object is not included. The firstrecognizing unit 22 stores all the values indicating the types of theobjects recognized from the images in the inspection section inassociation with the inspection sections in the storage device 12. Thefirst recognizing unit 22 reports all the values indicating the types ofthe objects recognized from the images in the inspection section to theselecting unit 23. The first recognizing unit 22 calculates therecognized frequency of each of the recognized objects and may reportthe frequency to the selecting unit 23. The first recognizing unit 22does not divide each image into a plurality of regions, extracts thefeature amount from the whole image, and recognizes the object byinputting the feature amount vector with a feature amount as an elementinto the general object identifier.

Based on the object recognized from the images in the inspectionsection, the selecting unit 23 selects the sub-category to which theobject included in the input video belongs. The sub-category is acategory obtained by categorizing the categories with a plurality ofobjects identified by the general object identifier used by the firstrecognizing unit 22 into a plurality of categories. That is, thesub-category is selected from among the plurality of sub-categoriesincluding some objects from among the objects belonging to the category.Furthermore, the selecting unit 23 acquires the certainty degreeindicating that the object included in the video is included in theobject of the selected category. The selecting unit 23 may categorizethe input videos into the groups corresponding to the selectedsub-categories from among the groups indicating the genre of the videocorresponding to each of the plurality of sub-categories.

The sub-categories are set to be different from each other at least onepart of the belonging object for each sub-category. For example, a firstsub-category related to ball sports and a second sub-category related tomotor sports are set in advance. In this case, the objects belonging tothe first sub-category includes, for example, “ball,” “person,” and“grass”. On the other hand, the objects belonging to the secondsub-category includes, for example, “car,” “motorcycle,” and “road.” Thesectioning of sub-category may be performed in a point of view differentfrom the genre of the general images such as sports and dramas. Forexample, a sub-category in which the object (for example, such as anocean and a specific logo mark) that may be included in a specific sceneand a sub-category to which the objects that may be included in anotherscene may be set.

The selecting unit 23 inputs the information indicating the recognizedobject into the group identifier to select the sub-category. Theinformation indicating the recognized object is, for example, a pair ofvalues indicating the types of the recognized objects. The groupidentifier outputs the identification number and the certainty degree ofthe sub-category according to the input. For example, “1” is set to“person,” “2” is set to “car,” “3” is set to “ball,” and “4” is set to“sky.” If the recognized objects are “person” and “sky,” “1” and “4” areinput into the group identifier. Alternatively, the selecting unit 23may input the value indicating whether each of the objects belonging toa predetermined category is recognized by the general identifyingprocessing and a vector included as an element into a group identifier.For example, if the value of the element is ‘1,’ the element indicatesthat the corresponding object is recognized, and if the value of theelement is ‘0,’ the element indicates that the corresponding object isnot recognized. For example, as described above, the objects belongingto a predetermined category are “person,” “car,” “ball,” and “sky,” and,the elements of the vector to be input into the group identifierindicate whether the objects “person,” “car,” “ball,” and “sky,” arerecognized in this order. If the recognized objects are simply “person”and “sky,” a vector (1,0,0,1) is input into the group identifier.

The group identifier is formed by learning with teacher, for example.For example, the group identifier works as a machine learning systemsuch as a decision tree, the multilayer perceptron, the support vectormachine, or the k-nearest neighbor method. For a sub-category, aplurality of pairs of numbers indicating the types of the objectsbelonging to each category is prepared as sample data used to learn thegroup identifier. When the pair of numbers indicating the types of theobjects corresponding to the specific sub-category is input into thegroup identifier, the identification number of each sub-category isoutput. In this manner, the group identifier learns by using a learningmethod according to the machine learning system for forming the groupidentifier.

The group identifier is formed to have a frequency for each of therecognized objects as an input. In this case, the sample data used forlearning of the group identifier includes the pair of the numbersindicating the types of the objects and the frequency of the object.

By additionally inputting a pair of values indicating the types of therecognized objects or the frequency for each object into the groupidentifier, the selecting unit 23 acquires the identification number andthe certainty degree of the category to be selected.

For example, if the group identifier is formed in the multilayerperceptron, the output layer of the multilayer perceptron includes theoutput unit corresponding to the sub-categories, respectively. By usingthe value between 0 and 1, for example, the value output from eachoutput unit indicates the certainty that the object included in thevideo belongs to the sub-category corresponding to the output unit. Ifthe output value is larger, the possibility that the object included inthe image belongs to the selected category is higher. Therefore, thegroup identifier outputs the identification of the sub-categorycorresponding to the output unit that outputs the highest value as theidentification number of the sub-category to be selected. The groupidentifier outputs the highest value as a certainty degree from amongthe output values from each output unit.

The group identifier may include a support vector machine correspondingto each sub-category. In this case, the value of certainty output fromeach support vector machine is included in approximately −1 to 1, forexample. The value of certainty indicates a possibility that the objectincluded in the video belongs to a sub-category corresponding to thesupport vector machine. If the output value is larger, the possibilitythat the object included in the video is included in the selectedsub-category is higher. Therefore, the group identifier outputs theidentification number of the sub-category corresponding to the supportvector machine that outputs the highest value as the identificationnumber of the selected sub-category. The group identifier outputs thehighest value from among the output values from each support vectormachine as a certainty degree.

Furthermore, if the group identifier is a recognition model based on ak-nearest neighbor algorithm, the recognition model detects k sampledata positioned near the pair of the values indicating the type of eachobject recognized from the inspection section. In this case, k is aninteger that is equal to or more than 3. The group identifier selectsthe sub-category with the largest corresponding sample data from amongthe k sample data. From among the k sample data, the number of thesample data corresponding to the selected sub-category is m (m is aninteger that is equal to or lower than k), the group identifier outputs(m/k) as the certainty degree.

The selecting unit 23 stores the identification number and the certaintydegree of the selected sub-category into the storage device 12 inassociation with the input video. The selecting unit 23 sends thecertainty degree to the calculating unit 24.

Based on the certainty degree and the prediction value of the number ofappearance objects, the calculating unit 24 calculates an evaluationvalue indicating adequacy of the inspection section.

The prediction value of the number of appearance objects in thenon-inspected section in the video is acquired according to thefollowing formula, for example.

R(t,j)=S×(T−t)×A(j)   (1)

In this case, S indicates the number of images per unit time. And, Tindicates a time length of the input whole video. And, t indicates thetime length of the inspection section. Therefore, {S×(T−t)} included inthe right side of Formula (1) indicates the total number of imagesincluded in the non-inspected section. And, A(j) indicates the averageobject appearance number per image with respect to the sub-category jselected by the selecting unit 23. In this case, j is an integer from 1to M, and M is the total number of sub-categories. Furthermore, forexample, A(j) is calculated as the average value of the number ofobjects in each image included in the plurality of sample videosprepared for each sub-category and is then stored in advance in thestorage device 12. That is, according to Formula (1), when thesub-category j is selected as R(t,j) and the time length of theinspection section is t, the prediction value of the number ofappearance objects is obtained.

For example, according to the following formula, the calculating unit 24calculates the evaluation value F(t,j) in a case where the sub-categoryj is selected and the time length of the inspection section is t.

F(t,j)=D(t,j)+wR(t,j)   (2)

At this time, D(t,j) indicates the certainty degree calculated by theselecting unit 23 when the sub category j is selected and if the timelength of the inspection section is t. And, w indicates a weightingcoefficient. For example, w is set in advance in such a way that themaximum value of the second term of the right side in Formula (2) issubstantially equal to the maximum value of the first term of the rightside. Therefore, based on either a certainty degree D(t,j) or aprediction value R(t,j) of the appearance object number, the value ofthe evaluation value F(t,j) is prevented from being substantiallydetermined.

As expressed in Formula (2), if the certainty degree D(t,j) is high,that is, if the possibility that the object included in the input videobelongs to the selected sub-category is higher, the evaluation valueF(t,j) becomes high. If the prediction value R(t,j) of the appearanceobject number in the non-inspected section is higher, that is, if thenon-inspected section is longer, the evaluation value F(t,j) becomeshigh. If the prediction value R(t,j) of the appearance object number inthe non-inspected section is higher, that is, if the non-inspectedsection is longer, the evaluation value F(t,j) becomes high.

According to Formula (2), even if the certainty degree D(t,j) is low,the evaluation value F(t,j) may be relatively higher when the predictionvalue R(t,j) of the appearance object number is high. In this case,however, the selected sub-category may be wrong, and the object includedin the video input as a result does not belong to the selectedsub-category. If the selected sub-category is wrong, there is a highpossibility that the second recognizing unit 26 described below may notapply the appropriate individual object identifier to the non-inspectedsection. As a result, the object recognition accuracy in thenon-inspected section may be reduced.

If the certainty degree D(t,j) is smaller than a threshold value H, thecalculating unit 24 may calculate the evaluation value F(t,j) bysubstituting the low certainty degree into Formula (2) instead of thecertainty degree D(t,j). In this case, the threshold value H is set tothe average value of the minimum value and the maximum value obtained bythe certainty degree D(t,j), for example. The threshold value H maybecome the minimum value of the certainty degree D(t,j) of which therate with the wrong selection result of sub-category is higher than therate with the correct selection result. The low certainty degree valueis set to be equal to or lower than the lowest value, for example, −1 or0. The calculating unit 24 sends the evaluation value calculated by theabove-described processing to the determining unit 25.

The determining unit 25 determines whether or not the inspection sectionis appropriate based on the evaluation value F(t,j). For example, thedetermining unit 25 compares the evaluation value F(t,j) to thethreshold value L. If the evaluation value F(t,j) is equal to or largerthan a threshold value L, the determining unit 25 determines that theinspection section is appropriate. On the other hand, if the evaluationvalue F(t,j) is smaller than the threshold value L, the determining unit25 determines that the inspection section is inappropriate. For example,regarding to each image included in the non-inspected section in thetarget video, the threshold value L is set in such a way that theaccuracy for recognizing the object by using the specified individualobject identifier for each sub-category is set to the minimum value ofthe evaluation value that is higher than the accuracy for recognizingthe object by using the general object identifier. For example, theminimum value is acquired by acquiring the accuracy for recognizing eachobject by using the individual object identifier and the general objectidentifier with respect to the plurality of sample videos prepared foreach sub-category in advance. For example, if the certainty degreeD(t,j) and wR(t,j) are values within the range from 0 to 1, thethreshold value L is set to 1.5 to 1.8, for example. The determiningunit 25 outputs the determination result to the processor 13.

If the determining unit 25 determines that the inspection section isappropriate, the second recognizing unit 26 recognizes the objectincluded in the image by using the individual object identifiercorresponding to the selected sub-category regarding at least one imageincluded in the non-inspected section in the input video. The processingperformed by the second recognizing unit 26 is equivalent to theprocessing performed by the first recognizing unit 22 except the sectionthat includes the target object and the object identifier to be used.The individual object identifier used by the second recognizing unit 26will be described below.

Like the general object identifier, the individual object identifierworks as a machine learning system such as, for example, a multi-layerparceptron, a support vector machine, or a k-nearest neighbor algorithmthat is formed by learning with teacher. The individual objectidentifier learns in such a way that simply the objects belonging to thesub-category corresponding to the individual object identifiers arerecognized. That is, a plurality of sample images is prepared for theobjects belonging to each sub-category. It is preferable that aplurality of sample images includes no object to be recognized isprepared. The individual object identifier corresponding to eachsub-category uses the sample image prepared for each sub-category tolearn according to the method of learning with teacher corresponding tothe machine learning system for forming the individual objectidentifier. Therefore, the individual object identifier recognizessimply the object that belongs to the corresponding sub-category.Therefore, if the correct sub-category is selected, the recognitionaccuracy of the object by the individual object identifier is higherthan the recognition accuracy of the object by the general objectidentifier.

The second recognizing unit 26 reads the individual object identifiercorresponding to the identification number of the selected sub-categoryfrom the storage device 12. The second recognizing unit 26 divides eachimage included in the non-inspected section into a plurality of regions.The second recognizing unit 26 extracts at least one feature amount fromamong the plurality of regions for each image. The feature amountextracted by the second recognizing unit 26 may vary according to theselected sub-category. The feature amount extracted by the secondrecognizing unit 26 may be different from the feature amount to be inputinto the general object identifier. Therefore, the second recognizingunit 26 may use the feature amount, which is suitable to recognize theobject belonging to the sub-category, to recognize the object accordingto the selected sub-category. That is, the recognition accuracy isimproved. By inputting the feature amount vector with the extractedfeature amount as an element into the individual object identifier, thesecond recognizing unit 26 recognizes the object that is included in theregion for each of the plurality of regions. By inputting the featureamount vector with the feature amount as an element extracted from thewhole image, the second recognizing unit 26 may recognize the objectincluded in the image. The second recognizing unit 26 may recognize theobject regarding simply one or more images selected from the pluralityof images included in the non-inspected section. The number of images inwhich the object is recognized may be one among 10 to 30 images, forexample.

The second recognizing unit 26 stores, in the storage device 12, thetypes of all the objects recognized from each image in the non-inspectedsection in association with the non-inspected section of the targetvideo.

FIG. 4 is an operation flowchart of the image recognizing processingthat is executed by the image recognition device 1. The setting unit 21sets the inspection section to the target video (Operation Op. 101).

By using the general object identifier, the first recognizing unit 22 ofthe processor 13 recognizes the object included in the image for eachimage in the inspection section (Operation Op. 102). The selecting unit23 of the processor 13 selects one of the plurality of categories basedon the recognized object (Operation Op. 103). The selecting unit 23calculates the certainty degree D(t,j) with respect to the selectedsub-category.

If one of the sub-categories is selected, the calculating unit 24 of theprocessor 13 calculates the evaluation value F(t,j) based on theprediction value R(t,j) and the certainty degree D(t,j) of the number ofappearance objects in the non-inspected section (Operation Op. 104). Thedetermining unit 25 of the processor 13 determines whether theevaluation value F(t,j) is equal to or larger than a threshold value L(Operation Op. 105).

If the evaluation value F(t,j) is smaller than the threshold value L (Noin Operation Op. 105), the setting unit 21 extends the inspectionsection simply by a predetermined additional section (Operation Op.106). After that, the processor 13 repeats the process after OperationOp. 102. However, in the second processing or later in Operation Op. 102executed, the first recognizing unit 22 may recognize the objectregarding simply the image in the additional section.

On the other hand, if the evaluation value F(t,j) is equal to or largerthan the threshold value L (Yes in Operation Op. 105), the inspectionsection is appropriate. By using the individual object identifiercorresponding to the selected sub-category, the second recognizing unit26 of the processor 13 recognizes the object included in the image foreach image in the non-inspected section (Operation Op. 107). After that,the processor 13 ends the image recognizing processing.

As described above, based on the result of the object recognition withrespect to the image in the inspection section, the image recognitiondevice selects one sub-category from among the plurality ofsub-categories that includes some of the objects belonging to apredetermined category. By using the individual object identifier thatidentifies the object belonging to the selected category, the imagerecognition device recognizes the object included in the image in thenon-inspected section in the video. Therefore, the image recognitiondevice may reduce a possibility for wrongly recognizing the object thatdoes not have a possibility for recognizing each image in thenon-inspected section is not included. Therefore, the recognitionaccuracy of the object is improved. The image recognition device maylimit the object to be recognized with respect to each image in thenon-inspected section. Accordingly, the calculation amount of the objectrecognition may be controlled. The image recognition device corrects theinspection section based on the evaluation value that is calculatedbased on the certainty that the object included in the video isselected. As a result, the image recognition device accurately selectsthe sub-category and extends the section in which the individual objectidentifier is applied.

The present invention is not limited to the above-described embodiments.For example, according to a deformation example, the individualrecognizing unit may re-recognize the object included in the image byusing the individual object identifier corresponding to the selectedsub-category. Therefore, the image recognition device may improve therecognition accuracy of the object with respect to the image in theinspection section.

According to other deformation examples, if the determining unitdetermines that the inspection section is inappropriate, the sectionsetting unit may change the position of the inspection section thatoccupied in the video instead of extending the inspection section. Forexample, if the determining unit determines that the inspection sectionis inappropriate, the section setting unit changes the position of theinspection section in such a way that the start time of the inspectionsection that is sequentially set is the end time of the currentinspection section. Furthermore, according to another deformationexample, if the position information of the place included in the imageis added to each image of the video, the section setting unit may setthe image, which corresponds to the place positioned within apredetermined distance range from the standard position specified inadvance, in the inspection section. The predetermined distance range isset to a range of 10 km to 1 km corresponding to a facility, forexample, a park, a station, or the like.

Furthermore, according to another deformation example, the evaluationvalue calculating unit may have the above-described certainty degreeD(t,j) or the prediction number R(t,j) of the appearance object in thenon-inspected section as an evaluation value. If the certainty degreeD(t,j) is the evaluation value, it is preferable that the inspectionsection setting unit firstly sets the inspection section to the shortestsection and that the determining unit extends the inspection sectionevery time the determining unit determines that the inspection sectionis inappropriate. If the inspection section gets longer, the number ofobjects recognized from the inspection section is increased. Therefore,the certainty degree D(t,j) becomes higher. Therefore, by graduallyextending the inspection section, the image recognition device mayproperly set the length of the inspection section.

The computer program that makes the processor execute functions of theunits included in the processing device may be provided to be recordedin a storage medium such as an optical medium or a magnetic storagemedium.

All examples and conditional language recited herein are intended forpedagogical purposes to aid the reader in understanding the principlesof the invention and the concepts contributed by the inventor tofurthering the art, and are to be construed as being without limitationto such specifically recited examples and conditions, nor does theorganization of such examples in the specification relate to a showingof the superiority and inferiority of the invention. Although theembodiments of the present inventions have been described in detail, itshould be understood that the various changes, substitutions, andalterations could be made hereto without departing from the spirit andscope of the invention.

1. An image identifying device comprising: a setting unit which sets asection having at least one image from among a plurality of imagesincluded in a video; a first recognizing unit which calculates aplurality of feature amounts related to at least the one image and whichacquires a plurality of identification results corresponding to each ofthe feature amounts from an identifier which may identify a plurality ofobjects belonging to a first category; a selecting unit which selects,based on the identification results, a second category having someobjects from among the plurality of objects belonging to the firstcategory instead of a third category having other objects with at leastone object different from the some objects; and a second recognizingunit which calculates another feature amount related to an imageincluded in another section different from the section and acquiresanother identification result corresponding to the feature amount fromanother identifier which may identify the objects included in the secondcategory.
 2. The image identifying device according to claim 1, whereinthe selecting unit inputs a determination result acquired from theidentifier into a category identifier which selects one of a pluralityof sub-categories that includes the second category and the thirdcategory, and wherein the selecting unit selects, based on an outputfrom the category identifier, the second category which includes anobject with a high possibility of being included in an image in theother section.
 3. The image identifying device according to claim 2,wherein, regarding each of the plurality of sub-categories, the categoryidentifier calculates a certainty degree indicating a degree ofpossibility included in each sub-category in which the object includedin the plurality of images and outputs the second category having thelargest certainty degree, and wherein the selecting unit selects thesecond category which is output from the category identifier.
 4. Theimage identifying device according to claim 3, further comprising: acalculating unit which calculates an evaluation value indicating anadequacy of a length of the section based on the certainty degreeacquired by the selecting unit from the category identifier and on aprediction value related to the number of objects included in an imagein the other section; and a determining unit which determines whetherthe length of the section is appropriate based on the evaluation value.5. The image identifying device according to claim 4, wherein thesetting unit extends or change the section if the determining unitdetermines that the length of the section is not appropriate, andwherein the second recognizing unit acquires another identificationresult corresponding to the other feature amount if the determining unitdetermines that the length of the section is appropriate.
 6. The imageidentifying device according to claim 4, wherein the calculating unitincreases the evaluation value as the certainty degree is higher or theprediction value of the appearance object number is higher.
 7. The imageidentifying device, comprising: a processor to execute; setting asection which includes at least one image from among a plurality ofimages included in a video; calculating a plurality of feature amountsrelated to at least the one image; acquiring a plurality ofidentification results corresponding to the feature amount from theidentifier which is able to identify a plurality of objects belonging toa first category; selecting, based on the identification results, asecond category having some objects from among the plurality of objectsbelonging to the first category instead of a third category having otherobjects with at least one object different from the some objects;calculating another feature amount related to an image in anothersection that is different from the section; acquiring anotheridentification result corresponding to the other feature amount fromanother identifier which is able to identify the some objects includedin the second category; and a memory which temporally stores theplurality of images.
 8. The image identifying device according to claim7, wherein, regarding each of a plurality of sub-categories with thesecond category and the third category, the processor calculates thecertainty degree indicating the degree of possibility that the objectincluded in the plurality of images is included in each of the pluralityof sub-categories, and wherein the processor selects the second categorywhich has the largest certainty degree.
 9. The image identifying deviceaccording to claim 8, wherein the processor acquires an evaluation valueindicating an adequacy of a length of the section based on the certaintydegree and on a prediction value related to the number of objectsincluded in the image in the other section, and wherein the processordetermines whether the length of the section is appropriate based on theevaluation value.
 10. The image identifying device according to claim 9,wherein the processor changes or extends the section if the length ofthe section is determined to be inappropriate, and wherein the processoracquires another identification result corresponding to the otherfeature amount if the length of the section is determined to beappropriate.
 11. The image identifying device according to claim 9,wherein the processor increases the evaluation value as the certaintydegree is higher or the number of objects indicated by the predictionvalue is larger.
 12. A storage medium storing image identificationprogram that causes a processor to execute operations, the operationcomprising: setting a section which includes at least one image fromamong a plurality of images included in a video; calculating a pluralityof feature amounts related to the one image; acquiring a plurality ofidentification results corresponding to each of the feature amounts froman identifier which is able to identify a plurality of objects belongingto a first category; selecting, based on the identification results, asecond category having some objects from among the plurality of objectsbelonging to the first category instead of a third category having otherobjects with at least one object different from the some objects;calculating another feature amount related to an image in anothersection that is different from the section; and acquiring anotheridentification result corresponding to the other feature amount fromanother identifier which is able to identify the some objects includedin the second category.
 13. The image identification program stored inthe storage medium according to claim 12 makes the processor execute thefollowing operation in the selecting processing, the operationcomprising: regarding each of the plurality of sub-categories with thesecond category and the third category, calculating a certainty degreeindicating a degree of a possibility that an object included in theplurality of images is included in each sub-category; and selecting thesecond category having the largest certainty degree.
 14. The imageidentification program stored in the storage medium according to claim13 makes the processor execute the following operation, the operationscomprising: calculating an evaluation value indicating an adequacy of alength of the section based on the certainty degree and on a predictionvalue related to the number of objects included in the image in theother section; and determining whether the length of the section isappropriate based on the evaluation value.
 15. The image identificationprogram stored by the storage medium according to claim 14 makes theprocessor execute the following operations, the operations comprising:changing or extending the section if the length of the section isdetermined to be inappropriate; and acquiring another identificationresult corresponding to the other feature amount if the length of thesection is determined to be appropriate.
 16. The image identificationprogram stored in the storage medium according to claim 14 makes theprocessor execute the following operation, the operation comprising:increasing the evaluation degree as the certainty degree is higher orthe number of objects indicated by the prediction value is larger.